1. Field of the Invention
The present invention relates to a distance information estimating apparatus which can estimate the distance from at least one image.
2. Description of the Related Art
A multi-viewpoint image includes a plurality of viewpoint images captured at the same time by a so-called camera array (known as camera array system, multiple lens camera, and the like), a plurality of viewpoint images captured by sequentially moving a viewpoint (an image capture position) of one still camera, or a plurality of viewpoint images created by sampling a video image captured by a video camera. In a case of calculating distance information to an object included in the viewpoint images, or generating an image of an optional viewpoint including a viewpoint which does not carry out image capture, parallax information corresponding to a viewpoint interval is necessary.
In order to extract parallax information from a multi-viewpoint image, there has been developed a multi-viewpoint image processing apparatus which obtains corresponding information of individual pixels in two images which are optionally selected from among a group of viewpoint images.
In the multi-viewpoint image processing apparatus, with regard to a certain pixel (a target point) on one captured image, a corresponding pixel (a corresponding point) is specified on the other captured image. A parallax about these pixels, that is, a displacement amount between a position of the target point in one image and a position of the corresponding point on the other image corresponding to the target point is calculated as the parallax.
Here, a matching process for specifying the corresponding point on the other image for the target point on one image between two images has various methods. For example, there is an area-based method of specifying the corresponding point by using a pixel block which is constructed by a plurality of pixels including a target point on one image as a template, and determining a correlation between the template and each of areas on the other image. Further, there is a feature-based method of extracting a feature which includes a target point on one image to specify the corresponding point on the other image on the basis of the feature.
In any of the area-based method and the feature-based method, a search area is set on the basis of a coordinate of the target point on one image, a correlation degree is determined about each of candidates of the corresponding point in the search area, and a pixel having the highest correlation degree is selected as the corresponding point, on the other image. In a case where the camera posture or a distortion is known, it is possible to estimate the depth of the corresponding points according to a triangulation method.
Further, there has been proposed a technique of estimating the distance from a picture composition of one image. For example, the depth can be estimated by analyzing a feature amount of the image, for example, a blur degree and the picture composition, and associating the feature amount with previously set depth information.
According to a method described in US 2009/0052796 A1, the feature-based method and the area-based method are mixed, the feature point is extracted, and a small pixel block (hereinafter, referred to as a patch) is generated around the feature point. The small patch is sequentially generated in the vicinity of the feature point by performing sorting out of the patch in which the patch is adopted, in a case where the patch can be viewed from a plurality of cameras. As a result, precise distance information estimation can be carried out.
According to a method described in Japanese Patent Laid-Open No. 2008-123019, the feature-based method and the area-based method are mixed, one of a plurality of images in which the camera positions are known is set as a standard image, and a two-dimensional mesh constituted by a plurality of triangles is generated on the standard image. It is possible to estimate the distance information of the image by determining the distance of each of apexes of the mesh so that the two-dimensional mesh converted into a viewpoint of the reference images other than the standard image coincides with the reference image most closely.
According to a method described in Japanese Patent Laid-Open No. 2012-029168, it is possible to estimate the distance information of an image by analyzing an image picture composition even from one image and collating it with a previously prepared depth structure pattern.
However, the methods according to US 2009/0052796 A1 and Japanese Patent Laid-Open No. 2008-123019 have a problem that the parallax of a flat part of the image can not be determined.
In the method according to US 2009/0052796 A1, the corresponding patch to the patch as seen from the other viewpoint can not be uniquely defined in a flat part which is a certain degree larger than the magnitude of the patch.
In the method according to Japanese Patent Laid-Open No. 2008-123019, the corresponding point to each of the apexes of the mesh as seen from the other viewpoint can not be uniquely determined in a flat part which is a certain degree larger than the magnitude of the mesh.
Since the method according to Japanese Patent Laid-Open No. 2012-029168 estimates on the basis of the previously prepared structure patterns, it is hard to carry out an estimation having a high precision, for example, a fine irregularity.